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ABSTRACT 

In adult educa'-ion, there are seven essential 
activities if evaluation is to have maximum use; understanding, 
specifying, describing, comparing, judging, valuing, and influencing. 
It is also necessary to understand the objectives being 
evaluated — their level (agency, program, project or course, and/or 
instructional) and kind (output objectives or objectives which help 
secure output) ; their charart eristics (results of decisions, 
discriminatory, predictive^ pluralistic, realistic, etc.). Objectives 
must also be understood as a system or parts of a system (content, 
behavior, and experience) and as having a role and function in 
programing (relationship to program purpose, source of the objectives 
and degree of commitment, meaning of objectives to those involved, 
guide or contract, etc.). The main problem in evaluation is getting 
good objectives written and used effectively in programing because of 
failure to deal with objectives as decision-making or failure to 
adjust concepts to overcome erroneous assumptions. The prospects for 
evaluating the attainment of objectives as a tool in programing 
depend upon adult educators building a concept of objectives and 
their use which is adjustable in adult education situations. (A 
fourteen- page bibliography accompanies the monograph.) (AG) 
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FOREWORD 



Sara Steele is rapidly emerging as one of the most knowledg- 
able people in the area of evaluatiorj in adult education. She is 
on the forefront of thinking ia the current developi?.,2nts in the 
contemporary approaches to evaluation, as she has ably demonstra- 
ted in a publication, Contemporary Approaches to Program Evalua- 
tion: Implications for Evaluating Program for Disadvantaged 
Adults , which she prepared for the ERIC Clearinghouse on Adult 
Education earlier this year. 

In the present monograph, she has teamed up with Robert 
Brack, who is also well versed in the latest developments in 
evaluation, to examine the process, properties, problems, and 
prospects in evaluating the attainment of objectives in adult 
education. 

The authors have noted that evaluation by objectives is a 
valid and useful, but limited form of evaluation. They have 
pointed out the advantages and problems of this approach to 
evaluation, thereby providing some practical guidelines for the 
adult cducaiOi". 

The special feature of this monograph is found in the in- 
sights which both authors have contributed to this approach by 
objectives previously considered by many educators as the best, 
if not the only effective kind of educational evaluation. 

The ERIC Clearinghouse on Adult Education presents this 
monograph as part of its belief in and commitment to improving 
the field of adult education. We are grateful to Sara Steele and 
Bob Brack for their contribution to this endeavor, as well as to 
Ms. Doris Chertow and the Syracuse University Publications in 
Continuing Education for making this publication more widely 
available . 

Stanley M. Grabowski 
Director 

ERIC Clearinghouse on Adult Education 
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INTRODUCTION 



Our society places extensive value on being able to set 
ai!d attain uujcccivcS, ^Jijch a process ic stressed in ednr^ition, 
business, and technology. Skill in judging the extent to which 
objectives have been attained is valuable in (1) providing feed- 
back to the learner, (2) guiding the teaching-learning transac- 
tion, and (3) evaluating instruction. It is an important ele- 
ment in evaluating programs to the extent that it helps identify 
key results. This skill should a^d the programmer in specifying 
targets and designing programs effective in reaching those 
targets . 

Of the many evaluation strategies available to adult edu- 
cators, the attainment of objectives model has received the 
greatest emphasis. The process seems simple and straightforward. 
Yet recent literature g.ves as much attention to the problems 
encountered as to the examples of success. 

This monograph is divided into four sections. The first 
describes the process of evaluating the attainment of objectives. 
The second summarizes some of the properties of objectives that 
must be understood if v;e are to successfully evaluate their 
attainment. The third reviews problems involved, and the fourth 
identifies prospects for this approach to evaluation. 
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THE PROCESS OF EVALUATING THE ATTAINMENT OF OBJECTIVES 



The process described in this section differs somewhat 
from the descriptions found in adult education literature. It 
ut/lizes soTiB recent conceptual developments related to evalu- 
ation and describes the process in more specific detail. It 
also puts as much or more emphasis on the use of information as 
on the procedures for securing that information. 

This section includes three main parts: (1) an example, 
(2) an outline of the process, and (3) comments on selected 
parts of the process. 

An Example of An Evaluation 

(Letter s>Tnbols are used so that the reader isn't distracted by 
program content. They can be replaced by content from your 
field, or with the following: 

X - supervision in an adult education agency; Y = administration 
of the agency; Z = staff and line position in the organization 
chart; A = morale; B = communication; C = production.) 

The statement of objectives for the training session and 
the description of what constituted attainment read as follows: 

As a result of the program, participants are to: 

1. Understand the role of X. In order to understand 
that role, participants mu^t be able to: 

a. Explain how X is related to V. 

b. Show the relationship between X and Z. 

c. Show the effect of X upon A, B, and C. 

2. Increase their positive orientation toward X. Such 
positive orientation will be demonstrated by respond- 
ing favorably to five key descriptions of X. 

3. Be able to use X in an everyday situation. Use will 
be determined by the skill with which the participant 
is able to complete a simple exercise requiring the 
use of X. 

Resources permitted the use of systematic research-type 
techniques so that the following description of results was 
available ; 

When the participants were compared with a 
matched control group it was found that signifi- 
cantly more participants than control group members 
(.01 level) were able to explain the relationship 
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of X to Y and the effect of X upon A, B, and C. 
Although stnnc had decidedly developed a *iore posi- 
tive attitude toward X, the difference between 
the participants and the control group was not 
great enough to rule out chance occurrence. Sixty 
per cent of the participants were able to der.ion- 
strate use of X in a simple exercise. The 
difference between the participants and the control 
group reached the .10 level of statistical significance. 

Working with this set of objectives and this description 
of results, the evaluation was as follows: 

This workshop was only fairly successful in 
attaining its objectives . For the most part , it 
resulted in the kind of understanding needed. 
Although it was not able to help many of the 
participants relate X to Z, this relationship 
really isn't crucial to the use of X. Its success 
in bringing about a clear understanding of the 
relation of X to A, B, and C was much more important. 

However, although the workshop was able to 
build important understandings, it was not able 
to influence the majority to take a more favor- 
able attitude toward X. And, although it did 
result in more of the participants being able to 
use X than would be found generally among those 
who did not take part in the workshop, 40 per 
cent still failed to master this objective. 

The basic question in this evaluation is 
whether or not the fact that the 40 per cent are 
unable to use X is serious. Can and should every- 
one be using X? In this case, because of X's 
contribution and its uniform applicability re- 
gardless of the situation, failure to aid the 40 
per cent is a iUajoL* problem on the part of the 
training session. This is particularly serious 
because the program was designed to meet a 
problem cc^used because too few of the participants 
were using X effectively. The inability of the 
workshop to develop a positive attitude toward 
X and the number who still can't use X effectively 
would mean that the one workshop has not been 
adequate in dealing with the causal problem and 
that additional work should be done. 

It is recommended that the programmers make 
every effort to continue contact with the twenty 
non-adopters and to provide whatever assistance is 
needed to bring the total adopter group up to 90 
per cent of all participants. It is also recotr- 
mended that the programmers continue contact with 
all participants through newsletters using examples 
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of succc'Stiful Li^io of X, values t'rori using X, and 
otluT niuL i vat ioiial. duvictis t*or reinforcing a 
positive attitude Coward X. 

Review of time usage within the program showea 
that relatively little time was given to the sec^ad 
and third objectives. Seventy-five per cent of the 
time and creative effort was spent on objective 
number 1. Tlie plan should be revised so that ob- 
jective number I is covered more efficiently and 
a greater amount of attention given to objectives 
2 and 3. 

Even thougli this workshop was not completely 
successful, it is to be noted that compared with 
other similar programs, this workshop was somewhat 
better in producing results consistent with Ity 
objectives than were similar workshops held last 
year in a similar cominunity. 

Activities and Tasks in the Evaluation Process 

The preceding example includes more than the usual eval- 
uation report. This evaluation starts after most reports stop. 
But it is the follow- through that gives the evaluation meaning 
and value. 

There are seven essential activities if evaluation is to 
have maximum use. Those activities are; UNDERSTi\NDINC , 
SPECIFYING, DESCRIBING, CObU^ARING, JUDGING, VALUING, AND IN- 
FLUENCING. Technically, only comparing, judging, and valu- 
ing are evaluation. The other activities are shared with other 
programming processes, but they are included in the following 
outline of process to show the complete Interrelated system. 

Each activity has a group of specific tasks. These tasks 
relate to each other and build the evaluation system. As a 
group they define the interactive process of evaluation. 

Activities and Tasks which Make up the Process of 
Evaluating the Attainment of Objectives 

UNDERSTANDING: 

Establish the purpose of evaluating the attainment of 
objectives. 

Define what good quality evaluation is in relation to 
that purpose. 

Determine who is expected to use the results of the 
evaluat ion . 



Identify what they need to know about the objectives 
and their attainment and when they need to know it. 



Determine the roles of participants, program 
personnel, admin i r/U or s , and ouher relevant 
individuals and groups in using the results. 

Identify the kind of strategy needed to be able 
to influence others to use the findings. 

Establish the function and role of objectives in 
the educational activity. 

Establish the intent, type, and meaning of the 
objective^. 

Determine the source of the objectives. 

Determine the extent to which the emphasis is on 
group attainment of the same objectives or indi- 
vidual attainment of objectives important to the 
individual , 

Determine whether objectives have changed during 
the course of the program. 

Determine the degree of precision appropriate to 
the quality and role of the objectives 

SPECIFYING: 

Determine the level, quality, and general clarity 
of the objectives. 

Translate the objectives into a list of examinable 
outcomes which describe when the objectives are met. 

Determine whether the objectives are of equal im- 
portance. If not, establish the ordering or 
weighting system that will be used. 

Select the outcomes that will be examined in the 
evaluation , 

Specify the standards that must be net by indi- 
viduals if an objective is attained. 

Specify group standards if they are appropriate. 

List outcomes other than those identified in the 
objectives — both positive and negative — which 
might logically be expected from the educational 
activity. 

DESCRIBING: 

Decide on the quality of evidence of results 
deemed necessary. 



Secure the appropriate information using approaches 
suitable to the purpose of the evaluation. 

Describe the outcomes that finally occurred from 
the program, both positive and negative, those 
expected in the objectives, and other results that 
occurred . 

Describe other factors which could have contributed 
to the results in conjunction with the educational 
activity . 

comparing: 

Periodically compare the results produced with 
expectations and standards as the educational 
activity progresses, and monitor progress. 

Compare the results at the completion with the 
objectives and standards. 

Compare the results produced with the originating 
need for the program. 

Compare the results produced with results produced 
by similar programs and by alternative approaches. 

JUDGING: 

Decide whether r,he results produced are sufficient 
to meet the expectations set in the objectives and 
standards . 

Decide whether any additional positive results that 
occurred are of sufficient value to "trade off" for 
lower attainment of the expected objectives, or 
whether they are "bonuses" to the expected results. 

Decide whether negative results occurred which are 
serious enough to over-ride the attainment of the 
objectives. 

Decide whether resources were used efficiently 
and effectively. 

Decide whether the objectives and standards were 
realistic and appropriate. 

Decide whether the educational activity did as 
well as could realistically be expected under the 
circumstances within which it was operating. 

VALUING: 

Determine whether the degree of attainment of the 
objectives and auxiliary results adequately 
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completed the purpose of the program and whether 
the need for the program was met. 



Determine the amount of value to the participants, 
community, and agency that the attainment of the 
objectives yielded; the amount of harm if objectives 
were less than attained. 

Identify the value of the educational activity in 
securing the results. 

INFLUENCING: 

Plan and carry out any necessary follow-up related 
to the educational activity. 

Make recommendaLJ.uns , as appropriate, to the 
participants, programmers, administrators, oThers 
doing similar programs, and/or tha public. 

Implement strategy that will influence the accept- 
ance and implementation of the recommendations. 

Add what was learned from the evaluation to 
understanding of effective programming and 
evaJ.ua t ion . 

Some of these tasks have been discussed extensively in the 
literature; others have received little or no attention. There- 
fore, we have more guides as to how to carry out certain tasks 
than we do for others. The reference list at the end of the 
paper is organized by activities so that those interested in a 
particular aspect can follow up in greater detail. 

However, evaluation is much more a "thinking" than a "do- 
ing" process. It is basically an intellectual activity. Its 
accuracy depends upon the objectivity, v;lsdom, and experience of 
those involved in the evaluation. Quality data can help make 
the task easier, but unless someone is able to interpret and use 
those data accurately they have little value. 



Comments on the Seven Essential Activities 

There's much that could be said about each of the activi- 
ties and each of the tasks. Our comments are limited to points 
we feel are particularly crucial in applying the concept of 
evaluating the attainment of objectives to programs for adults. 

Understanding 

Good evaluation requires an understanding both of what's 
to happen in the evaluation and an understanding of the objec- 
tives. Keys to understanding objectives will be presented in 
the second section. Here we*ll concentrate on understanding 
what's happening in evaluation. 
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Good evaluation accomplishes something important. It is 
done for a purpose. The focus of that purpose might be the 
participants, program personnel, or funders and other influen- 
tials. The evaluation must be suited to the prime audience and 
the purpose in relation to that audience. It is judged in terms 
of the extent to which it is important > timely, acceptable, and 
useable by its primary potential users. 

If, for example, as is often the case in adult education, 
the prime purpose of evaluating the attainment of objectives is 
to help adults learn and feel greater security in the learning 
situation, the process must be conducted in such a way that the 
learners aren't threatened and will accept and use the results 
of the evaluation to improve their own behavior. They must get 
the results in order to use them. Some evaluation approaches 
never share the results with the participants. 

If the prime purpose is to help programmers learn more 
about programming and improve educational activities, it must be 
conducted in such a way that it is meaningful and accepted by 
them. 

If the prime purpose iri to build greater appreciation for 
the program on the part of legislators or other inf luentials, it 
should deal with what they most want to know about the attain- 
ment of objectives. 

One way to better understand what needs to be done and to 
prepare the way for the evaluation to be utilized is to share 
major decisions about the evaluation with the prime audience 
for that evaluation. If the purpose is to help learners pro- 
gress, involve them in determining what objectives are most im*- 
portant and what constitutes the completion of an objective. 
If the purpose is to help programmers or inf luentials, encourage 
them to define exactly what they want to know about the attain- 
ment of those objectives. If the purpose is to secure inform- 
ation for your o\m use, decide what you want and why you 
want it. 

Unfortunately, in some instances (proof for funding) the 
process of evaluiation has to be remote from the people who are 
really involved in order to be credible. However, if that is 
not the reason why you are evaluating, explore ways in which 
those who must act on the results can share in decisions about 
standards, evidence, judgments, and value, and in carrying out 
the various evaluative activities. Shared evaluation is very 
compatible with the less formal student-teacher relationships 
essential in adult education. It can be an extremely useful 
tool to both the participant and the programmer. 

Specifying 

Philosophy and procedures for carrying out the tasks in- 
volved in specifying have shown considerable development in 
general education during the past ten years. Many of these 
ideas can be very useful in adult education if the adult 
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educator ts able to discern the extent to which his situation is 

the same as that in wliich chti ideas wero developed. If he finds 

differences, he must ha able to adapt the ideas until they are 
valuable tools In his own situp.tion. 

One difference in "^^onie piograns is the learner ^s concern 
about what he Is to leai:;. (.lUihirfU are often considered pas- 
sive beings without a rir;ht to decide or the wisdom to make the 
right decisicn about what b'tiuuld rt^'Sult from learning. Adults, 
on the other hand, ut,ually have some kind of broad end in mind 
when they participate in learning activities. 

Adult edfjccators need to set and evaluate the attainment of 
objectives Important to adults. Such objectives usually are 
fairly nebulous and global. They may be broken down into spe- 
cific tangible outcomes, but each specific usually has little 
value if disassociated from the whole. For example, knowing how 
to play the scale is of little value to an adult who wants to 
play a popular song. He will tolerate the scale as a means to 
the larger end, but he would not want to stop just with playing 
the scale. In adult education it is particularly important that 
the specific outcomes used to guide instruction and for collect- 
ing evaluative data be combinable into more global objectives — 
recognizablG to the adult as important. Being skilled in the 
specific behavioral outcome approach to objectives is not 
enough. If that approach is used, you have to be able to build 
more important behaviors from those specifics. If, as in many 
cases, you start with more global objectives, you must break 
them into specific evidences of their attainment. In either 
case, you should be able both to reduce to essentials and to add, 
in order to achieve combined importance. 

Turning each specific outcome and the standard for its 
attainment into an objective works well when the end that is to 
be attained can be easily divided into independent parts. How- 
ever, when a complex or indivisible behavior is involved, it is 
better simply to list under the objective the variety of things 
acceptable as evidence of attainment. For example, the follow- 
ing types of activities might be specified as acceptable evi- 
dence of attainment of an objective of increasing leadership: 

1. Running for an elected office. 

2. Serving as an ofUccr of a club or organization in 
che community . 

3. Serving on a committee, study group, or work group 
concerned with the betterment of the community. 

4. Actively attempting to Influence others in regard 
to a particular issue in the community. 

5. Showing support and cooperation to elected officials 
by supporting their stands in discussion, or carry- 
ing out tasks requested by the official. 



ERIC 



6. Carrying out projects or activities for the 
bettermunt of the community. 

7. Aiding a group or community through the solving 
of a problem, counseling of an issue, etc. 

Are some of the items on the list better evidence of 
leadership ability than others? Would you expect a person to 
demonstrate all of them or at least four of them before you felt 
he was showing leadership? 

A second major part of specifying is determining what con- 
stitutes adequate performance. The most usual types of deci- 
sions deal with (1) quality of the performance, (2) degree of 
correctness, (3) number of behaviors if the objective involves 
a process or set of possible responses, and (4) essential be- 
haviors if some are more essential than others. Several dimen- 
sions are involved: 

1. When has an individual learner achieved a 
particular objective? What standard will 
specify what must be achieved? 

2. When has the individual learner achieved ade- 
quately on a set of objectives? How many 
objectives must have been attained? 

3. How many learners must have achieved an 
objective for that aspect of the educational 
activity to be considered a success? 

4. How many objectives must a certain percentage 
of the learners attain for the activity to be 
successful? 

Individual Learners . In adult education It Is essential 
that standards be realistic in terms of the learner. How often 
have you heard a participant say, "Teacher says it should be 
like that, but this is good enough for me"? Standards that are 
too high may make the whole teaching seem artificial and unreal 
to the learner and turn him off. Valuable learning time may be 
wasted in "ever-teaching" for higher performance. 

Being realistic is a problem regardless of whether the 
teacher, the student, or the teacher-student together set the 
standard of performance. One must deal with such questions as, 
"WTiat quality of performance is it realistic to expect from this 
adult who comes from this kind of life situation and has this 
amount of time and energy to invest in this learning activity? 
What level of performance is absolutely essential? Is there a 
difference between the essential and the realistic?" If there is 
a difference, the teaching will have to motivate the learner 
sufficiently or the program is doomed to less than full success 
before it begins. 
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Not only must decisions be made about standards for each 
objective, but when an educational activity has more than one 
objective, the degree ot attainment of the various objectives 
that is considered success has to be determined. Let's say 
there are five objectives. The participants have adequately 
attained three out of the five. Would you conclude that the 
program did attain its objectives? Some would say no — each one 
must be adequately attained. Others would say yes — the majority 
of the objectives were achieved. Others would stress the fact 
that the two most important ones had been achieved and then 
would present plausible reasons for why the lack of success on 
the other two weren't serious flaws in the program. 

Group Attainment . Next, one has to deal with the question 
of how many participants must have attained the objectives for 
the educational activities to be considered a success. 

The traditional approach to education assumes that a given 
objective is equally appropriate and important to all learners 
and that the whole group should be trying to attain that objec- 
tive. In those instances where this philosophy is appropriate 
in adult education it is necessary to specify the number within 
the group that have to attain a particular objective and the 
number of objectives that have to be attained by that percentage 
of the group for the educational activity to be considered suc- 
cessful. Is 100 per cent required or is 80 per cent suffi- 
cient? Findings from practice adoption studies would indicate 
that for practices that are extremely difficult or which run 
contrary to habit and tradition, 20 per cent might be excellent 
at the beginning or end of the adoption cycle, depending upon 
the particular practice. 

When specifying the number of objectives that must be 
attained, one often responds in terms of a scale. For example, 
the program was successful in achieving its objectives (80 per 
cent or more of the participants attained all five objectives). 
The program was fairly successful in meeting its objectives 
(90 per cent attained two objectives, 80 per cent attained two 
others, and 70 per cent attained the fifth). The program was 
only moderately successful (50-60 per cent attained each ob- 
jective). Or the program fell far short of attaining its 
objectives (from 20-50 per cent attained the five objectives). 

The above example assumes that all of the objectives were 
of equal importance. However, a different scheme would be 
needed if two of the objectives were seen as being essential 
and the other three as being peripheral. Then 80 per cent for 
the two essential objectives and 50 per cent for the other three 
would be deemed a highly successful program. 

In most instances, judgments in terms of the mastery of a 
total pattern or the program^ s attainment of its whole package 
of objectives is more important than the degree of attainment 
of each as individual, independent parts. Knowledge about 
each, however, helps better identify trouble spots. 



ERIC 



11 



But the iipproaches common in traditional education may not 
be approprlattf In ddult education. Group attainment may be in 
conflict with a belief in individual differences and autonomy 
of the learner. If only one person out of twenty participants 
in a class on small engines needed to learn a certain technique 
specific to old-style lawn-mowers and he did master that ob- 
jective, this would mean moro In terms of it being a successful 
program than if nineteen people who never used a lawn-mower were 
compelled to master that particular thing. If one believes in 
adult autonomy, success would be defined in terms of the number 
of individuals helped to achieve their own individual objectives. 
Problems adult educators face in completing the task components 
of specifying will be discussed in the third section of this 
paper . 

Describing 

Some aspects of describing have h^id more attention in the 
literature than any of the other major evaluative activities. 
There are excellent texts of tests and medsurements and other 
aspects of research methodology which deal with the procedures 
of securing and analyzing data. Those processes are adjuncts 
within evaluation, necessary in proportion to the needs and 
resources of the situation, but do not comprise the main aspects 
of evaluation. If one feels compelled to list items such as 
design, instrumentation, and statistical analysis within thii 
evaluation process, they become substeps of the task of securing 
evidence . 

However, there are more fundamental considerations, lief ore 
you can decide upon a particular procedure for getting evidence 
and selecting the best methodology, you have to determine the 
type of evidence that is required and the degree of accuracy 
called for in order for the purpose(s) of the evaluation to be 
met . 

What evidence do you need that an objective has been 
attained? Is it sufficient for the individual to say that he 
has attained it? Or must he furnish proof — evidence of behavior 
in a real or test situation — that he has mastered it? Or must 
there be evidence that he is using it in hij. life situation? 
In some instances it is sufficient just to get learners' assess- 
ment of how much progress they have made with an objective. In 
others, you will want visible evidence of mastery. 

Can you rely on your own observations or must you in some 
way isolate and quantify the behavior? The extent of care used 
in getting either observational data or data by means of tests 
or other devices must be reconciled with the importance of the 
data and the costs to the participant and to the program which 
may arise from substantial data collection. Program costs in- 
clude both the money costs involved in processing the data and 
the time and student relations costs if the data collection 
interferes with the program or is burdensome or offensive to the 
student. The latter costs are key factors to be reckoned within 
adult education. Most programs try to accomplish too much in 
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too short a time. Taking a sizeable percentage of that time for 
non-teachlng — i.e., admiaistering instruments — creates a major 
loss unless the instruments directly contribute to learning. 
Secondly, adults will tolerate some data collection if they 
find it interesting or believe that it is valuable, but even 
the most cooperative adult has limits to the time and energy he 
is willing Lo spend on pencil work. Adult education is con- 
stantly challenged to find data collection methods which require 
minimum input from students and which are acceptable and inter- 
esting to the students. 

What constitute sound data about the attainment of objec- 
tives? The information mjst be (a) accurate, (b) bias free, 
(c) replicable, (d) germane to the objective, (e) communicable, 
(f) well-timed, and (g) credible to those who must use it. How 
the data are obtained is important only to the extent it influ- 
ences these five characteristics. The degree of the various 
qualities needed probably varies with the situation. There is 
no guarantee that extensively processed data (secured through 
complicated instrumentation and sophisticated statistical tech- 
niques) actually are any more accurate or less biased than 
expert observation. However, the quantitative data usually 
appear more "objective,^' replicated, and credible. They usually 
are more communicable than are individual observation and judg- 
ment. The process of carrying out quantitative data-gathering 
is more apt to reveal and expose biases. However, there may 
also be greater opportunity for other distortions and diversions 
to creep in. The time involved in processing often limits the 
timeliness of the information. It's ready after the decisions 
are made. The programmer or evaluator has to decide for a 
particular evaluative situation what method will give the most 
useable high quality data. It is seldom possible to get perfect 
"proof" of attainment of objectives. It is important to get the 
soundest data possible and to use these with care. Often some 
kind of compromise must be reached which balances the rigor 
needed with the resources that can be invested on the part of 
the agency, programmer, and participant. Cost-benefit analysis 
of evaluation is as important as is cost-benefit analysis of 
programs . 

Evaluation has been plagued with belief that an experi- 
mental design with pre and post-data collection and/or with a 
control group in which participant and control students are 
randomly assigned is the only bona fide way of getting data for 
an evaluation. Anything else was considered second best. Ex- 
perience is proving that the traditional methods do not often 
show a statistically significant difference and when they do, 
differences are challengeable on otlier grounds. Often such 
efforts are difficult to organize and prevent change in the 
program. Such designs lirit the use of feedback data while the 
program is in process. In the end it is difficult to tell 
whether or not the objectives of the program have been attained. 
(Traditional methods contribute information about the amount of 
learning, but judgments still have to be made as to whether that 
amount was sufficient in terms of the objective.) 



Experimental design and most other research techniques 
were developed in attempts to prove conclusively that a partic- 
ular phenomenon exists. In how many adult education situations 
is conclusive proof necessary, appropriate, and possible? When 
conclusive and unchallengeable data are reeded, the experimental 
design and rigorous instrumentation-analysis techniques should 
be used if there is reason to believe that they will not distort 
the proof to the point where it does not deal with the real-life 
program. 

In other instances, it may be more helpful and efficient 
to start with the aspect of judgment; then try to determine what 
quality data can be gotten within existing resources and how the 
limits of those data can be counterbalanced and corrected for. 
As an example, for some uses testimonial-type evidence may be 
fully as valuable or more valuable than rigorous test data, but 
care should be taken that it comes from a large enough and a 
representative enough sample to give a picture for the whole 
group. 

One of the harms of the past emphasis on research-type 
activities is that it has downplayed and obscured the importance 
of the programmer building the type of evaluation competency 
shown, for example, in judging diving competitions. The research 
approach replaces human skill with computer printouts. Adult 
education has not, as a field, been sufficiently counterbalanced 
with emphasis on building professional judgment. This trend 
must be reversed. In the long run, it's at least as important 
for the adult educator to be able to judge the attainment of 
objectives on the spot as he progresses with the teacher-learner 
interaction (i.e., through discussion^ work assignments in class, 
etc.) as it is for him to master research and statistical tech- 
niques. The information provided by such techniques and the 
techniques themselves are aids to improving his observation but 
should never be viewed as a replacement for it. Unfortunately, 
there is little in the literature about how accuracy can be 
developed in direct observation by the programmer. It should be 
one of his key professional tools in the same sense that diag- 
nosis is one ot the key professional tools of a doctor. 

When adult educators do use research techniques in eval- 
uation, they need to identify the kinds of adaptation that 
should be made. For example, statistical levels lower than 
.05 may be appropriate. In real-life situations where one is 
not trying to gather conclusive proof, a .25 level may be 
sufficient. And, related to adult autonomy and differences, in 
many cases it may be more meaningful to examine the number and 
per cent making gains rather than to focus exclusively upon 
group averages. 

Little attention has been given to the kinds of research- 
like processes which are best used in evaluation which has 
program improvement as its purpose. For the most part the rules 
of the game have been borrowed directly from research and deal 
with proof. Those rules may be inadequate and inappropriate. 
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In evaluation, for example, rather than concentrating all 
resources in a rigorous approach to one collection of evidence, 
it may be wiser to divide the resources among several approaches 
so that data about performance on an objective might be secured 
in two or three ways. Consistent data from several less rigor- 
ous sources may be aj rccurate as rigorously produced data from 
one source. 

ComparinR and Judgin g 

Although comparing and judging each has its own distinct 
tasks and qualities, they are usually done in tandem. Results 
are compared with the standards in the objectives and a judg- 
ment is made in terms of whether or not the objective was at- 
tained. Therefore, these two activities will be discussed 
together . 

They are alike also in that they have received less at- 
tention in the literature. Frimarily, they are alike in that 
they are carried out within the mind of the evaluator, and the 
process is difficult to describe or analyze. The accuracy of 
the judgment is dependent upon the accuracy of the comparison, 
and both are dependent upon the accuracy with which the eval- 
uator perceives the description of results and the objectives. 

Several kinds of comparisons are important in adult edu- 
cation. First, of course, is the comparison of the description 
of results with the objectives to judge if they were attained. 
Second, is the comparison of the results with the purpose or 
original problem or need frcm which the program arose to judge 
if by attaining the objectives, or perhaps, in spite of the ob- 
jectives, the program did what it was supposed to do. Third, is 
the comparison of the results of this program with results of 
similar programs and the judging of its comparative 
effectiveness . 

Judging Whether Objectives Have Been Attained . If the 
description of the actual results matches the stiindards set 
there is no problem. But what happens if it does not? For 
example: assume the program was to be considered successful 
if 70 per cent of the participants attained the five objec- 
tives, and the data show that only 50 per cent did. Was the 
program really less productive than it should have been, or wns 
there something in the situation that meant that a 70 per cent 
attainment was too much to expect? Suggesting this kind of re- 
examination poses some dangers. The temptation to rationalize 
away failures is high. However, in some adult education con- 
texts sufficient experience to set realistic levels may be 
lacking, and educators unfairly penalize their program because 
of this lack. Therefore, when actual performance does not match 
the ideal expected, the programmer has to explore which of the 
following factors may have been acting: (1) the data collection 
procedures were inadequate; (2) the levels were too high — analy- 
sis of the context within which the program was held or examina- 
tion of the participants and their life spaces may show that the 



objectives were unrealistic; (3) the program did not do enough 
related to the objectives. Retrospective analysis of the input 
to the program may shov; that, indeed, the input did fall short. 

Judging Accomplishment Against the Need . Results should be 
compared against the original purpose as well as against the ob- 
jectives. After all. the objectives are merely means to accom- 
plioiiing that original purpose and are not ends in and of them- 
selves. If they were well set in relation to the problem, need, 
or purpose, their attainment may automatically resolve the need. 
If not, a judgment still has to be made as to whether or not the 
program accomplished its purpose. Evidence in regard to the 
attainment of objectives may be strongest if the objectives are 
set aside and net considered in gathering evidence about the 
results of the program. The program and need, or general pur- 
pose from which the objectives developed, would then be the 
source of focusing evidence collection, and the first basis of 
comparison. If in a secondary comparison the findings showed 
great results in the areas specified in the objectives, this 
woi Id be stror^g proof not only that the objectives were attained, 
but that, they were on target in terms of the original purpose or 
need. The programmer was able to use objectives effectively in 
accomplishing the purpose. Using objectives as the focus for 
the evaluation may bias the examination toward those objectives 
at the very onset and provide more challengeable data. 

If the comparison with need or auxiliary examination of re- 
sults shows important outcomes besides those in the objectives, 
or instead of those in the objectives, judgments have to be made 
in terms of whether they are equally valuable and whether they 
should be counted in determining the success of the program. 

Examining participant satisfaction may also be important. 
Although the results d ita may indicate that the attainment does 
not meet expectations, the participants may feel perfectly satis- 
fied. On the other hand, results data may suggest that the ob- 
jectives have been attained — technically, the program is a suc- 
cess — but if the participants aren't satisfied the technical 
success is not a real success. 

Judging the Attainment Against that of Other Programs . Two 
important gains are made when it is possible to compare the 
attainment of a program with the attainment of other programs. 

First, such comparisons are extremely helpful in determin- 
ing what it is realistic to expect from programs with a given 
type of input and a particular type of clientele. It took ex- 
amination of several classes, for example, to decide whether it 
was realistic to expect improvement equivalent to a grade level 
for 200 hours of class time in adult basic education. Comparing 
a program with the best that has been attained in the past and 
by the average attainment of programs in this field gives a 
sounder means of judging its success than merely comparing it 
with paper standards. 
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Second, comparison helps determine whether other methods 
would have been as or more productive at lower cost. Assume 
that computer programming of instruction attains its objectives. 
If it proves more costly than programmed instructions and shows 
no other marked advantages, however, the comparison of the re- 
sults; of the two is as essential to decisions about using com- 
piuter instruction as is determining whether the computer can, 
in fact, produce the desired learning. 

However, compar^'son can be dangerous unless programming 
situations are identified. Care must be taken to see the ex- 
tent to which the participants held similar characteristics, the 
extent to which time and other resource utilization was the same, 
and whether there were any general factors in the context which 
would create different situations. 

Comparison is particularly important in adult education 
where individuality of programs and program settings often lead 
adult educators to continually reinvent the wheel (develop a 
program from scratch) without comparing that product with those 
of other inventors and adding the innovations of others that 
seem to pay off. 

Judging the Role of the Educational Activity . Educational 
activities can play many roles. Sometimes they provide new 
learning for adults. At other times little new learning occurs, 
but the learner is sufficiently motivated or reinforced so that 
he makes more use of what he already knew. This is often the 
case, for example, in nutrition education. Success lies not as 
much in gaining new understanding of the body's food needs, but 
in getting people to act upon that knowledge and eat the things 
they need but tend to avoid. 

The more basic question that adult educators sometimes 
have to deal with in terms of role is how much of what happened 
can be claimed by the particular program. There is a desire to 
prove to be the sole source. Yet, is this realistic in this day 
and age? Is it even wise in terms of what we know about how 
people are influenced to use knowledge? In many instances 
learners have access to information on a particular topic from 
several sources. It often is the reinforcing effect of the 
multiple exposure that results in the use of the practice or 
the change in the attitude, rather than the influence of one 
source alone. 

Improvinfi Comparison and Judgment s Comparisons can yield 
different conclusions. Two people making the same comparison 
can arrive at quite different judgments. Conclusions and judg- 
ments are debatable and should be questioned as a means of 
checking their soundness. A clear conclusion seldom comes 
directly and easily from the data. The programmer and/or 
evaluator must reach a conclusion based on logic as well as use 
cf the evidence. Both means must be able to support his con* 
elusion. They must be able to either refute or accept any 
challenges that may arise. However, such challenges are un- 
likely unless there is some kind of conflict element in the 
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situation. In such cases high quality evaluation is not enough. 
Success will primarily rest with the skill used utilizing the 
evaluation findings in arguing the case. 

It may be helpful to see how others interpret the data and 
what Judgments they form. Seeing what the person who is some- 
what negative to the program concludes as compared with someone 
who is sold on it, or as compared with two or three people who 
are uninvolved and neutral, will add perspective to the final 
Judgment. Discussing the findings with the actual participants 
also can be helpful in verifying the data and forming 
conclusions. 

Valuing 

Valuing is the process of interfacing judgments with rele- 
vant values and beliefs. It is the assignment o£ worth to that 
which was done. Valuing comes into play as the teacher examines 
the attainment of objectives and determines whether more teach- 
ing and learning experiences are needed. 

What does it mean to the participants if only 45 per cent 
mastered a particular objective? If the objective was important 
to them, then some follow-up may be needed to give further 
assistance to those who fell short. If, in a Basic Adult Edu- 
cation class, 45 per cent of the students did not master simple 
arithmetic needed in everyday living, it might be serious. On 
the other hand, if 45 per cent of those participating in a Great 
Books program failed to understand one of the authors, would this 
be serious enough to attempt to get them to expand their under- 
standing? Take the instance of the 45 per cent not mastering 
the simple arithmetic techniques. If further analysis shewed 
that the measurement techniques were fair and that 45 per cent 
had the capability to learn, then the programmer might want to 
seriously consider alternative approaches to teaching arith- 
metic or find ways of expanding che time allowed for slower 
learners and of giving them more individual attention. The 
teacher of the Great Books program might recognize that either 
insufficient time was given to the "troublesome" author, or that 
he had not been able to focus the discussicn sufficiently to 
help students grasp the main ideas. He would have to decide 
whether it was important to make changes the ne3Ct time he taught 
the course. 

More importantly, valuing comes into play in assessing the 
worth of what has been accomplished through attaining the ob- 
jectives. 

If the program is successful in helping participants 
achieve the objectives, how valuable is the result of that 
attainment? Is it really worth the cost in terms of time, 
energy, aiid money? Is the value sufficiently great that this 
program should be offer"*^ other people? That programming 
efforts in this area be expanded? These kinds of questions are 
more crucia,! than the preliminary question of whether or not the 
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objectives were met. A program can attain its objectives beau- 
tifully and still have little value. On the other hand, a pro- 
gram can fall short of its objectives and still be very valuable. 

It is important to put the whole thing together in terms of 
individuals and program. Putting it together in terms of indi- 
viduals means taking time to look at some of the class partici- 
pants as total persons and of seeing what these objectives, this 
performance, and these learning experiences have meant. Putting 
the program together as a whole means taking the findings from 
evaluating the attainment of the objectives and relating it to 
the program — going back over the program and examining the 
strength and quality of those inputs which were expected to be 
effective in helping to attain a particular objective. To what 
extent were the teaching and the learning experiences actually 
consistent with the objectives? What techniques appeared to be 
particularly effective with which types of students? What hin- 
dered other students in terms of a particular objective? Were 
time and energy effectively used in attaining the objectives? 
Were these the right objectives? 

Taking evaluation in its literal interpretation, it is the 
search for value or the means by which the worth of things is 
determined. If one really evaluates the attainment of objec- 
tives, then one is primarily concerned with determining the 
worth that completing those objectives has had to the partici- 
pants and to the greater society. 

Valuing culminates in recommendations for action. It is 
at this point in the evaluation process that exposure of beliefs 
and values exerts its effect upon the program. Understanding is 
needed of the beliefs and values which influence how various 
individuals and groups assess the objectives and their attain- 
ment. If values and valuing have been sufficiently dealt with, 
the influencing process which follows can be made more effective. 

Influencing 

Evaluating the attainment of an objective, as should be 
evident by now, is usually not an easy task. It can be done at 
varying degrees of precision. The evaluator must decide when 
such information is of sufficient merit tu outweigh the costs 
involved — to deal with such questions as: "What will I know 
if I do determine accurately whether the objective has been 
attained? How will I use that information? How useful will the 
information be? Is it worth more than other things I could do 
with that time and energy?*' 

It is pointless to expend all the effort just to prove a 
point. The conclusions should be carefully screened and used 
as a diagnostic device: (1) to design or recommend additional 
experiences for the participants if needed; (2) for improving 
the particular program examined; (3) for improving programming 
eftorts which expect a similar kind of result from a similar 
type of clientele. In addition to what is learned from the 
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findings of such an evaluation, there is usually considerable 
value in what is learned from the sometimes agonizing process of 
specifying the outcomes and setting appropriate levels of 
behavior. 

In ongoing everyday evaluation the person doing the eval- 
uation and the prime user of that evaluation are one and the 
same — the programmer and/or the learner. Then the question be- 
comes one of whether that individual can be honest with himself 
and accurate in his Judgments, and whether he will be willing to 
act upon those Judgments rather than to modestly deny that suc- 
cess exists or to rationalize away lack of success. 

Special efforts are required when others need to be in- 
fluenced through using the results of the evaluation. Informal 
as well as formal commun location of fi,nding3 and conclusions is 
important. But interaction and involvement are even more 
essential in helping those who need to act upon recommendations 
to utilize the findings of the evaluation in carrying out those 
recommendations. Not only must strategy be thought through 
carefully, but the tired programmer pressured by new things in 
the offing has to make time to complete his task in relation to 
the program he is evaluating. 
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PROPERTIES OF OBJECTIVES 



Objectives are a complex phenomenon. in order i:o use them 
well and to evaluate their attainment, one needs a clear under- 
standing of what they are and the role that they play in 
programming. 

Twenty years ago understanding objectives consisted pri- 
marily of a recognition that they were important and that they 
should have three parts — who, what, and how. There has been 
considerable work on the concept of objectives over the past few 
years and, as a result, our insight into the nature and role of 
objectives is increasing. 

This sectiou will summarize some of the comprehension that 
has been built up over the years and that the programmer must 
have before he can evaluate the attainment of objectives. 

The section is organized around fouL major kinds of under- 
standings necessary to effective evaluation; 

1. Undecstanding level and kind of objectives. 

2. Understanding characteristics of objectives. 

3. Understanding objectives as systems and par'Ls 
of systems. 

4. Understanding the role and function of objectives 
in programming. 

Understanding the Level and Kind of Objectives 
Where Evaiaation Will Be Done 

Evaluation of objectives is a generic activity. It should 
be undertaken for each of the kinds and levels of objectives 
utilized by the adult educator or adult education agency. Un- 
derstanding the differences in levels and kinds is important in 
developing evaluative strategy. 

Understanding Levels of Objectives 

Most adult education programs involve several levels of 
objectives. In even the simplest of situations, there are 
usually four such levels: 

1. Agency level of objectives describing the overall 
mission of the unit. This level is sometimes called "institu- 
tional goals," or comes from enabling legislation. 
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2. program objectives which indicate the key focus of 
the total activities for a period ot time (usually one to five 
years) . 

3. Project or course objectives which indicate the out- 
comes expected from one fairly f-'"able segment of the program. 

A . Instructional obj ec ich guide the specific 

teaching-learning interaction ig, class period, activity 

within a period). Instructio io jectives are designed to 
communicate instructional decisions ; not curricuJ.um, institu- 
tional, or societal decisions. Curriculum or program objec- 
tives deal with the broad goals of the educational activity. 
Instructional objectives deal with the specifics that are to be 
mastered as a part of reaching those goals. 

Instructional evaluation focuses primarily at the instruc- 
tional objectives level (specific changes in knowledge, skill, 
and attitude). Program evaluation focuses on the project, pro- 
gram, and/or agency objectives to see whether the specific 
changes in knowledge, skill, and attitude are combining in such 
ways that desired results are occurring in real-life practice. 

Recently attention has turned to looking at the relation- 
ships among the various levels. One level is valid only if the 
assumptions made* at another level are accurate. All four sets 
have to be sighted on the same target before there is any chance 
that attaining the objectives at a lower level actually will 
contribute to attainment of the more upper-level objectives. 
For example, it is possible for a programmer to attain all of 
his teaching objectives without making any contribution to the 
attainment of t:he objectives of the program just as it is pos- 
sible to win a battle but lose the war. In the ideal situation, 
a program objective is the overall package which includes sev- 
eral instructional objectives. It should be clear how the 
individual instructional objectives relate to each other and in 
combination, achieve the larger course, project, or program 
objective . 

There may be a variety of sub-levels in between the four 
main "floors" in the pyramid of objectives. In many instances 
in adult education the structuring of relationships among ob- 
jectives isn't clear and equal-distanced, but is more like the 
spl-^t-level approach to a building with objectives of various 
levels of complexity strung together in an upward or downward 
fashion. An instructional objective in some programs is as 
complex as a project objective ir. others. Or in other in- 
stances, a project objective may be as simple and concise as an 
instructional objective. 

An understanding of the number of levels of objectives in 
a particular situation is important for at least two reasons. 
First, it is necessary to determine which level has most merit 
in evaluation in relation to the purpose of that evaluation. 
Second, the various levels should have a clear relationship to 
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each other. In order co really evaluate progress at one level 
the relationship to anotiier level must be understood. 

We can compare the program of an adult education agency to 
a semi-truck. That truck has certain major parts — the van, the 
cab, the engine, the wheels, and underpinnings* Each must be 
runctioning for the truck to carry out its mission. Most 
agencies have several major programs. Each needs to work if the 
mission of the agency is to be met. Within a particular part of 
the semi (section of the agency's programs), there are sub-units 
which must function for that part of the semi to operate. For 
example, within the engine there are several parts — generator, 
carburetor, etc. --all required to be in order for the engine to 
work properly. Within each section of the program there are 
courses or projects which need to perform well for the .section 
to be making maximum contribution to the mission. Then within 
the engine sub-systems there are a variety of gears, wires, etc. 
that must be operational in order for the sub-systems such as 
the generator or carburetor to function. Instructional objec- 
tives can be compared to these gears and wires. 

The gear or wire taken out of its place in the generator 
or motor is useless. So an instructional objective which is not 
well-placed in the system that makes up the course or project 
objectives is of little value. Conversely, if crucial wires are 
missing or shorting, the generator can't start, the motor won't 
run, and the truck won't move. 

A mechanic checking over a semi on routine periodic in- 
spections, checks major units as well as wires and gears. So, 
as we think about evaluating programs, periodic evaluation is 
needed on all levels of objectives. Each type of evaluation 
may serve a somewhat different purpose. For example, evaluation 
at the teaching-learning level is extremely important to see 
that progress is being made on specifics. This kind of evalua- 
tion gives feedback in improving particular instructional activ- 
ities. But it probably does not serve well when trying to show 
a legislator that the mission of the agency is being achieved. 
The legislator is interested in composite results in terms of 
the objectives at the project, program, and agency levfil , and 
hov. much is occurring in terms of the mission for which the 
agency is funded. 

Traditional evaluation advice has been to reduce broad 
objectives, project, or program levels to their smallest com- 
ponent parts and concentrate the evaluation at that point. We 
come to recognize, however, that program evaluation and instruc- 
tional evaluation are not exactly the same thing. It becomes 
clearer that in program evaluation, although we reduce to 
specifics, we must be able to recombine those specifics to show 
the broader results described in the broader objectives. A 
watchmaker can test each part of the watch separately but still 
have no guarantee that the watch will run. It is the combination 
of the parts and their interaction that provide the accuracy 
upon which the watch is judged. 
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A watchmaker can evaluate the functioning of a watch with- 
out taking it completely apart. He knows what things to look at. 
So in many cases it is possible to measure the broader level of 
objectives without measuring each of the instructional objec- 
tives. If the purpose of a city-wide driver safety program is 
to reduce the number of improper turns being made into one-way 
streets, one can test the attainment of this objective by ob- 
serving turns at a sample of corners. It is therefore neither 
necessary nor appropriate to examine test scores on students' 
knowledge of proper turns or to measure attitudes on the im- 
portance of turning. The test of the objective rests in visible 
action. However, if there are still many errors in the counts 
at those crucial corners, then study of results on specific in- 
structional objectives may help identify where the driver edu- 
cation program needs to be changed. 

If you are concerned with project or program evaluation, 
check the instructional objectives against the project or pro- 
gram objectives to see if the necessary support framework is 
included in the instructional objective. Were the program and 
project objectives actually implemented in the specific learn- 
ing activities which were carried out? Sometimes the objectives 
detail only one part of what is included in the project or pro- 
gram objectives, like a tapestry which is only partially em- 
broidered. In good programming the completed parts (instruc- 
tional objectives) cover all parts that are essential for the 
visualization of the total picture (attainment of the project 
objective). Close examination sometimes shows, however, that 
the instructional objectives have deviated from the intent of 
the course, project, or program, and either fill in only the 
hazy and least relevant parts of the design or are part of 
another design entirely. 

Understanding Kinds of Objectives 

In an attempt to improve the attention given to results 
objectives, some sources have attempted to get programmers to 
ignore other types of objectives which function in adult edu- 
cation programs. In reality, the idea of an objective is 
generic, and there may be many types of objectives in program- 
ming, similar to the many types of wires in the semi-truck. 
Just as one wire doesn't replace another in the truck function- 
ing, so one kind of objective doesn't replace another in a 
program. Evaluation may be as important to one as to the other. 

Two broad categories of these types of objectives are 
output objectives and objectives which help secure output . 

Output Objectives . The two sub-types within output ob- 
jectives are: (a) Learner-centered objectives which Indicate 
what change is to take place in individuals as a result of 
participating (changes in knowledge, skill, attitude, or 
practice); and (b) Society-centered objectives which specify 
what effect the individuals who participate will have on the 
group. 
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Examples of objectives in the latter type might be; the 
community is to develop a recreation advisory group consisting 
of representatives of youth and adults; or, the amount of litter 
along the roadside will be reduced by 70 per cent. 

Bath types will be discussed in more detail in the next 
tow page^. 

Objectives Dealing with Securing Output . This group is 
less talked about. It encompasses a variety of kinds of ob- 
jectives. The most familiar are administrative or program- 
maintenance objectives and teaching objectives . 

Administrative or program-maintenance objectives are in- 
stitutional in nature, are concerned with program maintenance, 
or are instrumental objectives focusing on the mechanism which 
provides the program. This category includes objectives deal- 
ing with improved procedures, increased enrollments, improved 
contextual features, and personnel. Examples of such objectives 
are: to increase the number of Jow income participants; or to 
secure support for the workshop as a continuing activity; or to 
increase the number coming back for another program. 

The greater the "volunteer" nature of the audience, the 
more the program has to be concerned with setting and evalu- 
ating administrative objectives. Unless, of course, the agency 
does not care about its permanence and its growth and develop- 
ment. Some objectives in this category do deal with results — 
results in terms of the agency's well-being. It is important 
that attaitment of such objectives be carefully evaluated. It is 
possible to achieve program results in terms of the participants 
but not havti built adequate support among influentials for the 
agency to survive. These objectives should not be substituted 
for external results objectives, but often equal attention 
should be given to them bolh in programming and in evaluation. 

Teaching objectives state what the teacher is going to do. 
These objectives are usually the other side of the coin from the 
learner's objectives. Examples of such objectives include: 
to demonstrate the difference between an attitude and an opin- 
ion; or to stimulate students to explore on their own. Teach- 
ing objectives are useful to adulc educators. Evaluating to 
see whether or not they were carried out is also beneficial. 
However, results in terms of teaching objectives should not be 
equated with or substituted for examining the attainment of 
output objectives. For example, just because a teacher tried 
to stimulate students to explore on their own, does not neces- 
sarily mean that he was successful and that the student 
actually carried out more independent investigations. 

Programs probably will function most effectively if pro- 
grammers are able to use and to evaluate the attainment of 
various types of objectives. The important point , however , as 
mentioned above, is to avoid substituting instrumental objec- 
tives for output objectives. The achievement of instrumental 
or program-maintenance objectives should contribute to the 
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achievement of output objectives, but there is no real guarantee • 
that they do sd. B. 'cause it is often easier to examine results 
in terms of input or support objectives, and programmers often 
are forced to take the easiest way, for some years literature on 
evaluation has ignored anything other than output objectives. 
This may be both unfair and a hindrance to program development 
and improvement. Both types of objectives must be recognized as 
legitimate and complementary. Beth should be evaluated 
periodically . 

Understanding Characteristics of Objectives 

Most phenomena have distinctive characteristics. Some 
fairly common characteristics of objectives are beginning to 
emerge. Understanding them helps one to be better able to deal 
with objectives and their evalurition. Some characteristics and 
dimensions ot characteristics are discussed in the following 
sections : 

Decisions 

Objectives are the results of decisions. They communicate 
what one has decided to do. If written by the learner, they say 
what the learner has determined. If written by the programmer, 
they present tiie judgments of the programmer. 

If the learner and/or programmer can not make a decision, 
it is impossible to state an objective. Often either or both 
are able to make general observations but not to be specific. 
As a result, only general objectives are set without precise 
specifications and limitations. 

Program decisions require wisdom, experience, and suffi- 
cient knowledge to fed that one can s^t a decision that can be 
lived with. 

Discriminatory 

Pari: of decision-making is to choose among alternatives or 
to discriminate one thing from another. Objectives discriminate 
in the sense of setting certain things apart from others as 
being of most importance and thus receiving greatest attention 
and emphasis in the program. One has to be able to make choices, 
accurately compare values, understand the process of discrim- 
inating among those choices, and accept the consequences of 
that discrimination in order to set and use objectives. 

Prediction 

Attainment requires that the person setting the objective 
have the ability to predict what will happen or what can be 
caused to happen. 
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Commitments 



Objectives art* a coromitment cn the part of those who set 
them or accept them. Those involved must be willing and able to 
carry through on those commitments in order for the objectives 
to be meaningful V 

R>ealistic 

Objective:-: are most usetul when they are realistic. It 
has been said than the ultimate test of an objective is not its 
validity but its achievability . The identification of what is 
realistic is, of necessity, specific to each individual learner 
and specific programming situation. What is realistic in one 
situation may be impractical in another. 

Pluralistic 

The fact that objectives are not single entities but are 
usually pluralistic and hierarchical, and fit in a variety of 
patterns, is frustrating for the mind that likes to deal with 
sharp anO clear-cut entities. Accepting the amoeba-like rela- 
tionship of one objective to another, and disentangling the 
web to establish a viable pattern, is not an easy task. This 
becomes even more challenging as one deals with learners from a 
variety of backgrounds, as ife often the cate with adult learners. 
Very seldom are adult '^ducators dealing with simplistic changes 
in behavior unrelated to a variety of other behavioral patterns. 
This characteristic will be discussed in much greater detail in 
the next few pages. 

Challengeable 

Because objectives are the result of decisions and based 
upon tl\e values held by the person setting them, they are open 
to challenge and debate. In order to avoid such debate, ob- 
jectives are sometimes deliberately kept vague enough to be 
ablt* to mean different things to different people. If program 
goals are vague and intangible, it is possible to accommodate 
diveriT.e and inconsistent sub-goals within them. They facilitate 
flexibility while permitting goal succession and adaptation of 
working goals as the situation changes. They usually provide 
effective compromises without the work of forcing divergent 
factors to agree, and at times are the only way that a group 
can be moved into any kind of action. However, in addition to 
the dif f icultif^s of measuring and judging their attainment, 
vague objectives may lead to frustratic.i both within and without. 
They may produce role conflict and result "'.n deviation to unes- 
sential programs. Some programmers and in some situations, 
most astute programmers, will deliberately choose intangible 
goals rather than spes'^ific ones because of the political advan- 
tages involved in those goals. As a result, some of the least 
measurable objectives are written not by uninformed adult edu- 
cators but by those who are too knowledgeable and take the route 
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v'l zbc anb i>;u.'usiv st.it tM.i v^b. iectivo deliberately. The routt: is 
1. 1 ten r»>iK.wc'! bv t !,»- j t-rsoii wlio recogriizos the political over- 
Clones that <^CLur in :'.;.n;y ftiiuaLion situations. 

Changeable 

Another characteristic j^laguing ^valuation which requires 
understanding is change. The completion of one set of objectives 
and the need to move on to another may result in a program ending 
with goals different from the originals. In this case the 
changes should be made and accepted. Another type of change is 
that occurring when the original goals are displaced by others. 
Such displacement nay be appropriate if the situation changes or 
if the programmer becomes better informed about the situation and 
can be more accurate in setting goals. However, if the replace- 
ment results in sidetracking or investing extensive resources in 
non-productive objectives, the change is inappropriate. Some- 
times, for example, programmers might be blocked from achieving 
their original goals and seek out the easy things they can do. 
Or goals ot program and position maintenance become more impor- 
tant than goals of producing results, snowing that objectives 
may be only semi-stable should be useful in dealing with them in 
progran^ming . The programmer then has to determine when complete 
stability is appropriate and when some change is Dermitted. If 
change is appropriate, the evaluation design must recognize and 
take the change into consideration. 

Continuous 

Discussions of objectives usually cast them (intentionally 
or not) into a concept which has a starting point and a terminal 
point. However, in adult education some programming is contin- 
uous in nature with few terminations. Evaluation of such goals 
is not based upon completion but upon progress made between two 
points in time. Sometimes continuous objectives appear vague 
because they involve such large intents that they cannot be 
chopped up into specific time sequences. 

The more remote the goal the more difficult it is to deal 
with objectives as specific and concrete, and the more difficulty 
in measuring its attainment. Yet sometimes we may err in con- 
centrating too much of our evaluation efforts on examining what 
happens In terms of the short-range, concrete objectives, and too 
little in terms of the extent to which more remote objectives are 
being achieved. 

Some programs require remote or continuous objectives. Eval- 
uation then becomes a process of monitoring progress toward such 
objectives rather than establishing proof of their attainment. 
The extent to which a program is dynamic has to be identified and 
dealt with. 
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Enabler 



Objectives are communication links in a program. They 
communicate decisions and link program activities to the problem 
or other initiating reason for the program. Their attainment 
should result In the solution of the problem or the resolution 
of the purpose for the program. Their attainment is important 
only if the originating purpose is completed thereby. In this 
sense they are enablers. 

This is not an exhaustive list of the key characteristics 
of objectives in adult education. It does, however, provide 
examples of some of the kinds of major ideas that need to be 
listed and developed in order to help adult educators better 
use objectives. 

Understanding of Objectives as Systems 
and Parts of Systems 

Just as the botanist is better able to deal with a new 
plant when be knows the ways in which plants are generally cate- 
gorized and their characteristics, so the educator is better able 
to deal with an objective when he understands how it relates to 
other objectives. 

Approaching objectives through a concept of system can be 
helpf'-''- In that sucli a concept helps relate a specific objective 
to a spectrum of information. By placing the objective within 
that spectrum much more is known more rapidly than if it was 
seen as an unrelated object. 

All Objective as a System 

An objective is a system within itself. It has three in- 
terrelated parts: (1) rhe content or the "what" of the objec- 
tive; (2) the type of behavior or action expected in relation to 
that content or the "what*s he supposed to do with it" component; 
and the (3) "who" and "how many" coDjponent that indicates who's 
supposed to attain the objective. The three parts interact 
together and depict the "whole" of the learning that's expected 
to occur. 

In order to function efficiently all three parts must not 
only be present but must be consistont and in harmony wi^H 
other. They must be able to work together. For example, an ob- 
jective won't work if the behavior expected in relation to the 
content is incongruous with that content. An objective of people 
learning to sing a painting would be difficult to achieve; so 
would an nhjective to have learners make leadership. In addition 
to being in harmony with each other, both the content and the be- 
havior have to be in harmony with the characteristics of the 
learners. To expect beginners in a physical fitness class who 
have had no prior conditioning to run three miles the first day 
would be quite out of order. Within itself the three components 
of the objective must mesh and work smoothly together. 
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Each part of the objective is hooked into orher systems. 
For exan-ij-lc- . the content of a given objective hooks into systems 
for analyzing and categorizing types of content. Behavior ex- 
pected in the objective hooks into systems of behavior. The 
"who" and "how many" components hook into the socio-psychologi- 
cai, economic, and institutional systems that are individuals 
and groups. Being able to interpret a given objective in terms 
of how It fits into each of these various systems helps deter- 
mine both how to attain it and how to evaluate such attainment. 

Content Systems 

The content or "what" aspect of the objective varies ac- 
cording to the type of results. Included are such things as: 
{^a) specific subject natter in the case of objectives dealing 
with mastery of knowledge; (b) LliC nature of belief, idea, 
"object," etc., in objectives dealing with attitudes and values; 
(c) nature of the skill or attribute that is to be deve:^oj.>ed in 
objectives dealing with skill mastery or character development; 
and (d) description of the aspect in the environment to be 
changed if the objective deals with the change in conditions. 
The "what" aspect is generally the object of the verb in the 
sentence. It clearly defines what is going to be dealt with. 

Lach type of content has its own system for typing the 
generic nature of that content. Ascertaining the type within 
the content area helps determine how best to facilitate learning 
and development, how to measure amount of learning and develop- 
ment, and how to evaluate whether the progress is sufficient. 

Among the various types, greatest attention has been given 
to categorizing kinds of learning and development involved in 
cognitive learning. As you examine an objective in terms of the 
content being emphasized, determine whether the focus is on a 
fact. A relationship among facts? A concept? A principle? 
A process? Does it deal with specifics? If so, is the knowledge 
involved one of terminology? Knowing specific facts? Or is the 
knowledge knowing what to do with specifics — how to organize, 
study, judge, or criticize them? Are there conventions involved? 
Trends and sequences? Classif icat.'^ons and categories? Criteria? 
Methods of inquiry related to the specifics? If it focuses on 
universals and abstractions, does it focus on principles and 
generalizations? On knowledge of theories and structures? Are 
^her'? chains of specifics involved? Or if the focus is on con- 
cepts, is changing the concepts into a strategy required? In 
the series of questions above, did you recognize the categories 
of a taxonomy published some years back? Taxonomies provide 
systems frameworks for seeing how different types of content 
relate to each othct . 

The other areas arc less extensively developed. In the 
affective area a similar set of questions v;ould include: Does 
it focus on a belief? An opinion? A value? Is it a single 
entity or is it a part of a larger belief or value system? 
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If it is a skill is it fairly simple involving only a few 
muscles? Or is it complex? Does it involve only one part of 
the body or must several parts be co-ordinated? 

A parallel set of questions for environmental changes in- 
cludes; Is t'le development d«*.sired to be in an individual or 
of a group? Is the kind of < .ange desired in the environment a 
social change? an economic change? a physical change? a psycho- 
logical or eLidional change? a legislative change? an institu- 
tional change? 

The more precisely you can define the nature of the con- 
tent that is the heart of the objective, the more clearly you 
can determine how to deal with that content. Procedures and 
processes vary according to what is being emphasized. 

B ehavior Systems 

Just as a type of content is part of a system which shows 
what it is compared to other types of content, so the behaviors 
expected of the learner can be developed into systems which help 
to define what a particular kind of behavior is and how it re- 
lates to other types of hPhaviors. 

You need to know what the learne^ ;s expected to do with 
the cognitive content that is being dealt with. Is it to be 
repeated as learned? To be used in thoughts (associately) ? In 
solving problems (applicat ively) or in providing meaning to sub- 
sequent activities ( interpret ively) ? How far do you expect the 
learner to go In developing an ability to use the data? Is he 
simply to recall it? Is he to go farther and show that he com- 
prehends it? Can he apply it? Use it in synthesis? In analy- 
sis? In evaluation? 

If it is an affective content, a belief, value, or atti- 
tude, is he to be open to receive, be aware of what it may be, 
willing to receive, willing to concentrate on it? Should he re- 
spond? Assign value to it? Accept it? Commit himself to it? 
Should he integrate it into his value system? 

If it is a skill, is the learner to know what to do, ac- 
^^'dlre a set for what to do, be able to respond under gt??. dance, 
give correct response mechanically, or build the skill Into a 
habitual response? 

Does the behavior involved require recall (knowing what to 
do and when it*s completed)? Does it require manipulation (know- 
ing he.: to do it)? Does it require problem-solving (knowing how 
to match need and solution)? Is the behavior expected merely the 
knowing about doing somethir^g or does it require actual doing? 
If so, is the actual doing merely to prove the person an do it, 
or is he expected to integrate it into his regular lite behavior? 

Stages in practice adoption also are means of categorizing 
expected behavior of individuals and of groups. Is the indi- 
vidual or group only expected to be aware of a need for a change? 
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To test out ways of making the change and the effects of that 
change? To work to make the change a permanent one? 

Depending upon the philosophy of education that the adult 
educator is using, he either (1> has to be able to clearly de- 
fine the specific behavior that he as the teacher and/or pro- 
grammf^r expects to result, or (2) he has to understand the level 
and t>pe of behavior related to the content that the participant 
wants to achieve. In both cases he has to try to adjust those 
expectations to the realities of the situation. But by under- 
standing he can bring resources to bear in a much more direct 
and efficient manner than if he only has a vague idea of what 
needs to be done. 

Content and Behavior Combined 

Tables of specifications or outcome grids can be helpful 
in examining the meaning of an objective. Key elements of a 
content system are listed on one dimension of the table and the 
range of behaviors in the behavior system are listed down the 
other. Cells are then formed coupling behaviors with content to 
indicate the particular outcome selected for emphasis. Tables 
of specification are helpful in identifying the amount of empha- 
sis ar.d adjusting the emphasis of the evaluation accordingly. 

Experience Systems 

Most of the various contents and behaviors relate to each 
other in a type of sequential relationship. A person needs to 
knew certain things before he can make sense out of other things. 
He has to have developed certain abilities before he can go 
further with those abilities. One of the special challenges to 
the adult educator is that of identifying how the new objectives 
relate to past attainment. Does the adult have sufficient back- 
ground to cope with the objectives set for the particular sit- 
uation? Or has he already mastered many of those objectives, 
and does he need new or more demanding ones? This task is par- 
ticularly challenging when the adult educator or team within an 
agency does not have the same sequence of contact with the par- 
ticipant that is built up in an elementary or high school or 
within an apprentice or other program where the individual is in 
sustained contact for several months or years, and where inform- 
ation is readily available as to the kinds of previous experi- 
ences and attainment from those experiences. 

Relationships Among Objectives 

So far we have touched upon the individual objective as 
being a self-contained system and as hooking into other systems — 
content, behavior, and past experience. Another aspect of sys- 
tem important in understanding objectives is that of applying 
the concept to a set of objectives. Most projects and indi- 
vidual learning activities involve sets of objectives. Usually 
there is a relationship among the objectives within the set. 

o 
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They are not separate and independent. Or if they are treated 
as such, perhaps they shouldn^t be. 



Although cognitive, affective, and skill objectives are 
usually stated as separate objectives, attainment of one may be 
dependent upon the attainment of another. For example, adults 
don't usually bother to exert the effort necessary to master 
content or perfect skills unless they believe in what they are 
mastering. Conversely, it is hard to develop a positive atti- 
tude about something that you don't know anything about. Affec- 
tive, cognitive, and skill objectives have to complement each 
other and support each other toward attainment. 

Soma of the more typical patterns of relationships are: 
(1) simultaneous - progress toward two or more must occur at the 
same time; (2) sequential relationships (chains) - attainment of 
a latter oblective depending upon attainment of an earlier one; 
(3)hierarchical (pyramids) - one macro-level objective requires 
a host of micro-level objectives covering the total area in- 
volved in a macro-type objective; (4) essential (keystone) - one 
objective base upon which attainment of several other objectives 
rests . 

In evaluation we are usually attempting to assess the at- 
tainment of a set of objectives. It is important, then, to 
determine the kind of system involved In the set. If, for ex- 
ample, it is a keystone type, and if there are two or three 
basic outputs which if there guarantee that the whole set of 
objectives has been attained, rather than collecting data on all 
the set and its parts, the focus would be on the two or three 
essential outputs. Or, on the other hand, if there V7as clearly 
a sequential relationship among the objectives with one depend- 
ent upon the other, one might only examine the attainment of the 
final step of the sequence. Or if there was reason to doubt 
that the full sequence had been attained, one would attempt to 
develop a profile of how many successfully mastered each step 
in the sequence and where the major drop-off points occurred. 

Understanding of objectives as systems and parts of sys- 
tems helps one design the evaluation eystem most appropriate in 
a given situation. 

Understanding the Role and Function o£ 
Objectives in Programming 

In order to successfully evaluate the attainment of ob- 
jectives the evaluator needs to know both how objectives were 
expected to operate in the program and how they actually did 
operate. Some of the aspects that need to be understood as 
they operate within the given situation are: 

1. Relationship to Program Purpose 

2. Source of the Objectives and Degree of Commitment 
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3. Meaning of the Objectives to Those Involved 



4. Guidu or Contract 

5. Extent of Change Planned or Permitted 

6. Real or Rhetoric 

7. Program Focus or File Copy 



Relationship to Program Purpose 

Are the objectives considered ends in themdelves or are 
they seen as enablers to achieve the purpose of the program? 
If so, how well do the objectives cover that purpose? If the 
objectives are met will the purpose be attained? If yes, the 
evaluation may well limit itself to exp.mining the attainment of 
the objectives. If not, the evaluation may want to examine what 
happened in terms of the originating problem or need as well as 
examining what happened in terms of the objectives. 



Source of the Objectives and Degree of Commitment 

What was the actual commitment to the objectives on the 
part cf the program personnel and the learners? Were the pro- 
grammers committed to seeing that these particular objectives 
were achieved? Were the learners committed to achieving them? 

Just because objectives are written in terms of what par- 
ticipants are supposed to achieve does not necessarily mean that 
they are meaningful to those participants and that the partici- 
pants are commitned to tboc" particular things. 

In some situations the objectives actually are statements 
made by those in the program. However, in most instances they 
were developed by one or more of the following: representatives / 
of the expecte'i learners, some of whom actually participated and 
others who did not; individuals the programmer thought knew the 
expected clientele very well (for example, welfare workers rec- 
ommending objectives for educational programs for welfare 
recipients); representatives of the larger society or a partic- 
ular pressure group who wished to bring about certain 'changes by 
expected participants (for example, produce buyers who .wanted 
farmers to change their products); subject matter experts who 
expi'GSScd themselves in terms of what they felt others should 
knov* about their subject; a "think group" within an agency whose 
statement was based on the expected mission of the agency; or the 
teacher or program coordinator with past experience and contacts 
with one or more of the other sources. 
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There may be several different and conflicting sets of 
learner objectives functioning in a given situation. Some will 
be stated — others will not. The evaluation design has to deter- 
mine wh'^ther any objective© other than the "official ones" will 
be probed for and evaluated. 
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Meaning of the Objectives to Those Involved 



Words mean different things to different people. A key 
element In designing strategy for measuring and evaluating at- 
tainment of objective's Is that of determining what people think 
the objectives mean. Do the programmers and participants hold 
the same meaning? For example, an objective of helping families 
solve problems related to Income management could be interpieted 

the programmer as "providing some help on some problems/' 
while the participant might Interpret it as "solving all of my 
problems." If someone outside of those two groups is involved 
in the evaluation, he has to be sure that the meaning he assigns 
Is the meaning held by those in the program. 

Guide or Contract 

Are the objectives viewed by the programmer and the par- 
ticipants as "contracts" in the sense that there is a clear 
expectation on both parts that they will be attained pretty 
much as stated? Or are they intarpreted as guides offering 
satisfaction to programmer and participants if some progress is 
made toward reaching otherwise "final" objectives? In the first 
instance, programs would be judged according to the <cxtent to 
which the behaviors spelled out in the contract were in fact 
achieved. Less than complete achie^'ement would be faulted. In 
the latter case, complete attainment would be welcomed but not 
required. Substantial progress toward attainment of the ob- 
jectives would be accepted as success. 

There is a current emphasis in elementary education on 
performance contracting, with objectives being held as standards 
for full attainment. There may be many instances in adult edu-- 
cation where it is appropriate for the idea to carry over and 
be applied. However, fairly complete knowledge is required of 
what is realistic to expect learners to achieve from a certain 
amount of program input before one can set contractual-type 
objectives which have much hope of being attained. Elementary 
education has years of experience to draw upon. It also is deal- 
ing with a more homogeneous group of participants. In many in- 
stances, where such homogeneity and past experience is lacking, 
objectives may better be seen as guides, with success based on 
progress toward rather than full attainment. 

In addition to how the program personnel interpret the 
objectives » the evaluation may need to make a judgment in terms 
of whether there is sufficient knowledge about possible results 
in similar situations so that the objectives could and should 
be effectively used in the contract sense. 

Extent of Program Change Planned and Permitted 

Also important to the whole design of evaluation is to 
understand the extent to which the program is functioning in a 
pattern through which the basic decisions are made prior to and 
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outside of the learning interaction, i.e., are the objectives 
pre~sec In advance of actual program implementation? This is the 
usual concept of programming and the basis tor conventional dis- 
cussions of evaluation. However, if programming is seen as inter- 
active and dynamic, with objectives emerging as the programming 
proceeds, then rather than starting with a pre-plan with set ob- 
jectives, one has to log the decisions about objectives that are 
made as the program progresses. In such a situation, objectives 
may be short-range — emerge, be satisfied, and others emerge dur- 
ing the course of the interaction. Or objectives may be con- 
stantly interchanged — set, moved toward, and then replaced by 
others before they are completely attained. The kind of eval- 
uation of attainment of objectives that teams with teacher- 
student interaction may need to take a much more dynamic form 
than that which is coupled to a concept of programming as being 
the carrying out of a pre-set plan. 

In cases where the pre-set plan is important, how much 
deviation from the plan is permitted? Can objectives be changed? 
Did objectives change during the program and were those changes 
recognized? Or is the judgment to be made only that of whether 
the original objectives were attained? 

Real or Rhetoric 

Sometimes the set of objectives put on paper are developed 
as official statements designed to be pleasing to a funding 
source or outside public. They are not meant to spell out the 
actual specific goals toward which the program is working. A 
bit of probing may reveal that the printed statements are window 
dressing rather than the real expected outcomes. The question 
then becomes whether to proceed with the printed objectives as 
a basis of the evaluation in order to show their fraudulent 
nature, or to probe for the real objectives and evaluate the 
program in those terms. Usually the two sets are on the same 
track but involve different specifics. 

Program Focus or File Copy Only 

Did the objectives function only on paper or was there an 
effort to attain them? Did they provide a focus for all other 
decisions and was the program design specifically targeted to 
the objectives? Or were the objectives writti^n, filed, and for- 
gotten? A programmer cap believe that objectives should function 
as the central organizing focus in programming, but for a variety 
of reasons in actual practice, fail to operationalize that 
concept . 

If there is considerable evidence that the stated objec- 
tives of the program had little meaning in the real life situ- 
ation, is it worth while to try to evaluate their attainment? 
Might more be accomplished by retrospectively developing a list 
of realistic outcomes that probably were being attained by the 
program as it was being carried out? 
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PROBLEMS IN EVALUATING THE ATTAINMENT OF OBJECTIVES 



Certainly there are a variety of problems inherent both in 
measuring the phenomena involved in the objective and in making 
judgments as to the sufficiency of the attainment of the objec- 
tive. However, a more basic and overriding problem still hasn*t 
been solved — that of getting good objectives written and used 
effectively in programming. Until it is solved, there is little 
room for dealing with the problems specific to evaluation by 
objectives » 

Although evaluating the attainment of objectives has been 
the concept endorsed by the Adult Education Association of Amer- 
ica since 1952, there are few examples where the complete cycle 
has been effectively carried out. Evaluators who approach a 
project expecting to evaluate the attainment of objectives often 
report: (1) there were no statements of objectives or the ob- 
jectives were so vague they could be interpreted in many ways; 
(2) there was no agreement on and support of the official state- 
ment of objectives; or (3) the objectives changed during the 
course of programming. 

If clear objectives have been set and they have been used 
to guide programming operations, evaluating the attainment of 
those objectives is relatively simple. The major problem, 
however, rests with getting the obj'^ectives set and used. It is 
assumed that adult educators are both rble and willing to set 
objectives. Experience and analysis are showing, however, that 
this may be a much more difficult activity than appears on the 
surface. 

Part of the problem may clear up as we make more progress 
in understanding the phenomenon of objectives and being able to 
deal with properties such as those described in the second 
section. This section presents some additional ideas about why 
it is difficult to set and use objectives sufficiently to make 
evaluation of their attainment worthwhile. The ideas are or- 
ganized around two central themes: 

Failure to deal with objective setting as decision, 
making. 

Failure to adjust concepts to overcome fallacies 
in assumptions. 

Failure to Deal With Objective Setting as Decision-Making 

Objectives present the key decisions about the program^ 
Setting of objectives is the basic decision-making process for 
the whole programming operation. The writing of the objectives 
comes easily after (or if) key decisions are resolved. It is the 
resolving of those decisions that is difficult. 
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The Process of Setting Objectives 



Some approach objective-setting as though it is a writing 
rather than a decision-making activity and as a result may make 
little progress. Objective-setting tweeds to be seen as: 

1. A process of making choices among the various 
alternativas involved in a given program. 

— which clientele? 
— what content? 
— how much? 
— what results? 

2. A process of comparing the value of various 
alternatives in making the final choice. 

3. A process that requires criteria or a basis for 
choosing among alternatives. 

4. A process that requires highly developed ability 
to analyze, assign value, and make a choice. 
These are mental abilities and involve little or 
no physical action. 

5. A process which requires sufficient knowledge, 
wisdom, and experience to understand context 
and possible consequences of choices. 

6. A process of prediction and commitment. 

Objective-setting is much more than sitting down at a desk 
and writing out some objectives. The educator must be able to 
analyze in such a way that (1) abstractions and global dreams 
are reduced to their basic parts, the relationships of those 
parts are clear, and crucial elements are determined; (2) alter- 
natives are clearly identified; (3) the factors influencing the 
feasibility of the various alternatives are examined. Or, work- 
ing from a small piece or idea, he must be able to analyze the 
larger whole from which that part comes so that he sees the 
part in its perspective. He must be able to deal with the 
nitty-gritty of a real life situation and clear away the irrel- 
evant while concentrating on the essential. 

When the educator has a feel for the pieces involved, he 
must be able to assign value to those pieces. Will it be better 
to emphasize A or should be concentrate on B? He has to select 
items of the greatest value to the people who will participate. 
He needs the ability to reconcile conflicting values. Some 
prospective participants want a program which focuses on A, 
others want to explore B in depth. From the wealth of material 
that could be taught, he has to select what will be taught. Why 
are his selections better than his rejections? 

In other words, he is required to make decisions from 
among a wide array of choices and he needs some feeling that 
those decisions are right or at least are defensible. Usually 
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there are many ways in which a program could go. The decision 
is, which way should It go? 



He has to wrestle with these decisions sufficiently so 
that he can communicate them to other people. When he does, he 
is apt to get other suggestions and some challenge. Why does 
he feel the program should go in this particular way? There 
probably Isn't an absolute answer he can identify. 

Establishing Valu e 

It is impossible to set an objective if one is not willing 
to commit oneself to something as more worthy than something 
else. Yet many educators have had no formal training in valuing 
and comparing value. As one sets objectives, the question in- 
volved in making decisions is not whether a particular objective 
has merit but whether or not it is more valuable than some other 
objective . 

It is usually much easier for adult educators to choose 
among methods and techniques and to plan how they will carry out 
a program than it is for them to make the basic decisions 
(inherent in objectives) about why they are doing the program and 
what they expect to accomplish with the methods. Most of us know 
more about methods and techniques, and have broader factual and 
experiential bases for choosing among them than we do about set- 
ting objectives. The adult educator finds a wider range of 
literature to guide his choice of method than to guide his 
choice of objectives. 

For some years, about all the adult education literature 
emphasized in terms of guides for choosing among alternatives 
were the Tylerian concepts of sources of objectives (learners, 
society, and the discipline) and screening objectives through 
philosophy and psychology of learning. The latter part of the 
concept was often underdeveloped. 

Now additional ways of examining choices in objectives 
are being suggested. Criteria include acceptance — the degree 
of intellectual and emotional commitment that participants are 
apt to have in achieving the objectives; achievability — is it 
feasible to try to reach the objective; appropriateness — is it 
desirable to do so even though it may be feasible; worth — 
priority of importance as contrasted with other possible objec- 
tives. More recognition is being given to the idea that some 
objectives may not be attainable within the limitations of the 
educational setting. It is important to analyze barriers as 
well as needs when setting objectives. The fact that some 
achievable objectives may not be worth realizing also is being 
considered. There must be constant interplay and balance of 
two questions — what is it important to accomplish, and what 
is it most possible to accomplish? 

Although it is hard to judge objectives against such 
2>creens, the attempt may be useful in helping to isolate the 
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kind of decisions that need to be made and exploring alternatives 
related to them. And such screening of ideas for objectives may 
make the ret:ulting objectives more wcrthy of being evaluated for 
their attainment. 

Analysis, valuing, and decision-making all require exten- 
sive mental activity. For most of us this is extremely hard and 
slow work, with second and third thoughts involved. And yet we 
think we should be able to sit down and dash off useable objec- 
tives in an hour or two. 

Need for Building a Base for Choice 

Academic courses and in-service workshops often miss help- 
ing people set and use good objectives. We've passed the era 
when sessions on objectives concentrated primarily on insisting 
that teachers state objectives in terms of learner results rather 
than in terms of what the teacher intends to do. We have passed 
the era when time was wasted in circular arguments over the dif- 
ferences between goals and objectives. We have passed the stage 
when the emphasis was on the form of the verb and the concept 
that every objective had to have an infinite verb form (students 
to demonstrate). We are now at the stage of understanding the 
general categories of content and behavior that are usually in- 
cluded in objectives. This stage is certainly more helpful and 
productive than the past ones. However, too often, the emphasis 
is on helping adult educators with the mechanics of phrasing 
objectives rather than helping them screen their decision-making 
and evaluate the quality of the objectives that are stated. Too 
little experience is given in judging the value of alternative 
objectives and in helping the teacher get a feel for what is 
realistic, what can be attained by whom in what situations, and 
with what educational inputs. There is still a very long way to 
go in terms of developing learning experiences to help adult ed- 
ucators become such good predictors and selectors that they will 
feel secure in attempting to use objectives as the major means 
of evaluating their work. 

Bases for Sound Predictions 

As we realize that objectives announce the fundamental 
decisions about a program we may become uneasy, since we commit 
ourselves to the attainment of predictions. To make accurate 
predictions (make decisions about programs which result in clear 
and attainable objectives) the adult educator has to have a high 
degree of knowledge about (1) the program content; (2) the gen- 
eral characteristics of the learners* needs, interests, learning 
orientations, learning abilities, past experiences, and life 
situations; (3) educational processes and the effects of various 
types of learning inputs with his particular type of learners; 
and (A) specific knowledge about how the program content fits 
into the life situation of the learners and theii orientation 
and ability in the type of learning involved. 
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In order to specify standards of attainnient for those ob- 
jectives, the adult educator needs a sound understanding of what 
it is possible to attain with a given input in a particular sit- 
uation. Adult educators often differ from youth educators be- 
cause they are dealing with a much more heterogeneous clientele 
in terms of ability, past experiences, needs, and interest, and 
are dealing with situations where the life space of the indi- 
vidual is very apt to affect his learning. Therefore, some 
adult educators are put into the situation of needing to know 
more about the individual student than the elementary school 
teacher does but having less contact with the student and fewer 
records about him to draw upon. 

Predictions take on still another dimension in new programs. 
Some adult education programs are so Innovative that not only do 
they lack an experience track record on which to base predic- 
tions, but in some instances the programs are having to develop 
the knowledge as they proceed. There is no text or other ac- 
cumulated body of knowledge in the field. This is the case in 
many of the adult education programs which focus on real-life 
problems. Should the adult educator avoid such types of pro- 
grams? Should he limit himself only to those where he can make 
secure predictions and know that he can prove that those pre- 
dictions were reached (i.e., objectives are measurable)? Or 
should he be free to push the frontiers even though he may need 
to program for several months before he can gain sufficient 
knowledge to set any kind of predictable targets and levels for 
his objectives? 

When the knowl(?dge base is sufficient, the adult educator 
is more apt to be willing to accept and to enjoy the challenge 
of making the predictions of accomplishments required in ob- 
jectives and to accept the commitment to achieve those 
accomplishments. 



Failure to Adjust the Concept to OvercoJie Fallacies 
in Underlying Assumptions 

A second major reason why programmers may fail to set and 
use objectives effectively is that they are wise enough to 
realize that the narrow concept of objectives and their use as 
transferred directly from youth education may not fit their adult 
education situation. 

Any theory or process is based upon a set of assumptions. 
If those assumptions are faulty or do not apply to a given situ- 
ation, then the process itself is in difficulty. It may be that 
several of the assumptions underlying the ability to set and 
evaluate the attainment of objectives do not apply in all adult 
education situations. 

Here are some of the assumptions that may at times come 
into question: 

1. Enough is known by the field as a whole so that 
^ in all situations the appropriate result from 
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a given amount of Input can be predicted (I.e., 
it Is always possible to set clear outcome 
objectives) . 

The basic assumption Is that adult education always can 
set accurate objectives. In some fields there is sufficient 
past experience and research to be able to set a realistic 
target matched with a given amount of input. However, adult 
education is forever getting involved in pilot efforts and in 
new things and this assumption may not always hold true. 

2. The individual adult educator knows enough about 
the process of education and how his clientele 

:<pond and learn various types of things so 
^«iat he can set realistic objectives . 

The teacher who has taught grade 4 spelling for five years 
probably has a basis for making such a judgment. The beginning 
elementary teacher accepts the cumulative experiences of her 
thousands of colleagues who have been teaching Johnny to spell 
for one-hundred years. In such a situation, It is very probable 
that realistic objectives can be set. In some areas of adult 
education there may also be a wealth of experience sufficient to 
make possible setting realistic objectives which reconcile the ) 
amount of input with the wide range in the students. But in 
other areas the adult educator is constantly experimenting, sel- 
dom working from a prepackaged curriculum guide, and seldom do- 
ing exactly the same things more than two or three times in his 
professional career. Not only does he have fewer colleagues en- 
gaged in the exact same thing but his coiumunication with those 
colleagues may be much more restricted. He may not have a 
counterpart within fifty miles, while the elementary school 
teacher may have as many as twenty or two-hundred counterparts 
within the same city. 

\ 

In terms of knowledge or educational processes, adult ed- 
ucators have usually had little academic preparation for pro- 
gramming with adults. Few have majored in adult education. 
Most either have majored in some other form of education or in 
a specific program content field and then have gravitated to 
working with adults. 

3. It Is possible to predict and control adult 
learning in complex situations. 

Much of the type of learning involved in adult education is 
intricate and is related to the real world. The personalities j 
and life spaces of the adult participants are more complex than 
those of children. Transfer of the concept of attainment of ob- 
jectives to adult education assumes that it is equally possible 
to predict the achievable end accurately and to control events 
leading to that end in complex as in simple situations. 

4 . It is important chat decisions about results 
be made before teaching begins. 
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Many are teachers because they like the active interaction 
with learners. They are primarily actionists rather than plan- 
ners. They are impatient to get on with the action. Decisions 
are often made in the context of the learning situation and often 
on the spot as the interaction progresses. This leaves them free 
to capitalize on teachable moments. Longer-range goals and in- 
tents are present but are part of their subconscious and act in- 
stantaneously to guide the choice of emphasis or activity or role 
of the teacher in the interactive situation. 

Most formalized concepts of objectives consider them as 
"pre-active" entities — decisions made be.fore the action starts. 
There is an assumption that teachers cannot build integration, 
sequence, and concentration on major ends by utilizing only 
interactive decisions. Most concepts of objectives give in- 
sufficient attention to the dynamic situation where objectives 
are set and changed within the interaction setting and, cs a 
result, objectives often become something put on paper in the 
beginning but disregarded from that time on. They are not up- 
dated as the teachers' own decisions change through interaction 
with learners. 

Not only does the extreme actionist need to slow down suf- 
ficiently to be able to verbalize and communicate his subcon- 
scious goals and intents, but also the theorist working with a 
concept of objectives has to bridge the gap between the some- 
times sterile-looking statements required in plans and the 
dynamic oper;itlonal concepts involved in working objectives, 

5 . The teacher has sufficient control to produce 
the desired results . 

Another built-in assumption is that It is always humanly 
possible for the teacher to produce the expected results. The 
adult educator does not always have the same rewards and sanc- 
tions (grades and disciplinary procedures) to use as a strong 
impetus for adults to achieve objectives. Most adult education 
programs cannot require the same tractability as the grade 
school does, and as a result there is a high level cf unpredict- 
ability in adult education activities. There is a belief — held 
by the student if not by the teacher — in the autonomy of the 
adult and the adult's right to choose what he will and won't 
learn and what he will and won^t do with that learning. There 
are pressures in the grade school which enforce the "all will" 
concept underlying attainment of objectives. And a responsi- 
bility for socialization and prepaTation which makes those 
pressures appropriate. In most instances It Is inappropriate to 
apply these same pressures to adults. Pressure exerted through 
licensing and certification is justified, however, when the well- 
being and satisfaction of others who will depend upon the adult 
students services are involved. But in other instances the 
assumption that adult students always "will*' learn may be 
erronecuc. 



ERIC 



43 



^' !1''>-^'^1J1^ should l earn the same thinR . 

Most ot the views of adult education on evaluating the 
attainment of objectives have been borrowed from childhood edu- 
cation, where the teacher has a responsibility for social izatioix 
and for making sure that all (or as many as possible) of her 
students master certain fundamental skills. This concept of 
evaluation of instruction has been highly influenced by an essen- 
tialistic philosophv of education. There is a basic assumption 
that all participants shou.ld master the same objectives. In 
some adult education activities, particularly those where one 
has to certify students, this essentialistic view may be appro- 
priate. However, if one believes in all that we say about recog- 
nizing the variation in life space and experience of the adult 
and in autonomy of the learner — if one pays any attention to 
what research has rfhown about the range of reasons why people 
take part in adult education activities — we will question 
whether it is always appropriate to judge a program by the ex- 
tent to which it has helped all participants master the same 
thing. The life situations and needs of adults vary so greatly 
that sometimes a practice that may be excellent for ':he major- 
ity actually may be harmful for a few and of no value for a 
minority. For example, using behavioral objectives may be ex- 
cellent for adult educators teaching shop courses or math 101, 
may be harmful to those teaching philosophy, and may have no 
more value than otl.er approaches to those teaching current 
events . 

The adult educator has to become astute in deciding what 
proportion of his students can, will, and should master a par- 
ticular objective as he determines what amount of mastery is 
necessary before his work is successful. Interestingly enough, 
there*s some indication that childhood education may soon out- 
distance adult education in belief in and real operationaliza- 
tion of individualized instruction. There's a growing advocacy 
in general education for examining the extent to which indi- 
vidual children h:?ve attained objectives designed specifically 
for their situation and needs as opposed to or in addition to 
the extent to which members of a class have met a mass objective 
set for thc.t class. 

7 . \ttainment of Objectives Constitutes a Successful 
Adult Prcjgra m . 

In adult education, success lies as much or more in the 
satisfaction of the participant as it does in accomplishing 
tasks for society. In adult education more than in youth edu- 
cation, the test of the program lies in the extent to which the 
participant is satisfied with his experience. Although gener- 
ally satisfaction is related to accomplishment as indicated in 
objectives, this is not always the case. Objectives can be 
attained and participants go away unsatisfied. On the other 
hand, objectives may not be attained and participants may go away 
very satisfied if they have found other things of value to them 
within the experience. The extent to which the adult participant 
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is satisfied with the results is at least as important, if not 
mcv^ so, than the actual results produced. 

8 . Attainment o£ Objectives Automatically Produces 
Value? 

There is often an underlying assumption that attaining ob- 
jectives is good and automatically produces something of value. 
Worth comes from results of objectives only if the objectives 
are themselves valuable. Unless saved by serendipity, attaining 
mediocre objectives results in a mediocre project. A good rat- 
ing on value at the end of the project is dependent upon careful 
evaluation of the objectives as they are being set to see that 
they are indeed apt to produce outcomes of use to the participants 
and/or to society. The most Important evaluation of the program 
is its total worth and value. The attainment of objectives can 
be an enabler but isn't autoir^t ically so. 



ERIC 



45 



PROSPECTS FOR THE EVALUATION OF ATTAINMENT OF OBJr ^TIVES 



Although adult education holds evaluation of the attainment 
of objectives as an important part of evaluation, it should not 
be seen as constituting the whole of evaluation. It can be an 
extremely valuable tool for facilitating learning and improving 
programming when it. can be utilized without violating the basic 
assumptions upon which it is built and without jeopardizing the 
basic premises of a philosophy of adult education. 

Setting objectives is not easy. The process involves sub- 
stantial analysis, valuing, and decision-making. It requires 
broad knowledge and experience. Evaluating the attainment of 
those objectives depends a good deal upon the loi^ic and objec- 
tivity of the programmer and evaluator, although good quality 
data can help to improve the quality of the judgment. Despite 
difficulties, it is important tc do as much evaluation as is 
appropriate in order to gain greater insight into the effect of 
various program inputs. 

The last ten years have shown a great advance in the knowl- 
edge base for setting and evaluating the attainment of objec- 
tives, but there is still a long way to go. Even considering the 
problems discussed in the last section, we conclude that the 
prospects for evaluating the attainment of objectives as a more 
useful tool in programming are very bright. 

They are bright IF ADULT EDUCATORS ARf: ABLE TO BUILD A 
CONCEPT OF OBJECTIVES AND THEIR USE WHICH IS ADJUSTABLE IN ADULT 
EDUCATION SITUATIONS. The prevailing concepts fit certain adult 
education situations very well. It is a poor fit in others. 
Some of the kinds of adjustments that may be in order are: 

1. Adaptations to compatability with the philosophy 
of education being operationalized. For example,, 
if one is operating in terms of individual dif- 
ferences and adult autonomy and responsibility, 

the concept used of objectives and their evaluation 
should fit that philosophy. 

2. Adaptations to dynamic programming situations. For 
example, rather than to require preset objectives, 
objectives may be documented as they emerge and are 
dealt with in the course of the teaching-learning 
interaction . 

3. Suiting the guide vs. contract dimension to the 
degree of assurance with which results can be 
predicted. For example, if there's little past 
experience on which to base objectives, ^hey may 
be treated as hypotheses to be tested and pro- 
gressed toward rather than contracts cf absolute 
performance . 
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4. Increasing the kinds of guides available to adult 
educators as they make the decisions that result 
in objectives. For example, better outlining the 
kinds of decisions and the process of decision- 
making that ..2eds to occur. 

5. Better relating objectives to participant satis- 
faction so that the two elements are more con- 
sistent. For example, examining the extent to 
which participants are satisfied with the ob- 
jectives and the results as well as determining 
what the results were. 

6. Suiting the purity o^ method and the extensiveness 
of the data treatment to the consequences of in- 
accuracy . 

7. And through such adaptations, making objectives 
meaningful tools to adult educators in their own 
situations . 

Prospects are bright IF ADULT EDUCATORS ARE ABLE TO FURTHER 
DEVELOP THE BASIS FOR INTERPRETING DATA AND ASSIGNING VALUE AND 
FORMING JUDGMKNTS. To date, few evaluations have dealt with the 
question of when objectives have been attained. Few guides are 
given for developing accurate judgments. Most have been happy 
to prove that results related to an objective occurred. There 
needs to be considerably more exploration and discussion in 
terms of how one decides when an objective has been attained. 
There naeds to be more sharing of i ?sults so that "track records" 
are input for comparison. Most important, concepts of determin- 
ing th€". value of that attainment have had little development. 
Adult educators need to become much more involved in exploring, 
debating, and proving worth in programs. This may involve a 
good deal more than previously thought. 

Prospects are bright if adult educators CAN RELATE PROCESS 
TO PURPOSE /VND CAN DISCERN WHEN THE VALUE FROM EVALUATING ATTAIN- 
MENT OF OBJECTIVES WILL BE GREAT ENOUGH TO COMPENSATE FOR THE 
EFFORT EXPENDED. What are the "must situations" VKien some degree 
of attention must be given to the attainment of objectives? 'That 
are the "helpful s 5 ^.uations"? When is the process a waste of 
time? Part of the answer to these questions is determined by 
the extent to which the evaluation actually will be able to 
affect important decisions on the part of the learner, adult 
educator, and those who support the program. Part of the answer 
depends too on the utility of the evaluation and whether the 
evaluation was conducted in such a way as to enhance use. 

Evaluation has worth only if it is used. All types of 
evaluation, attainment of objectives included, should be used for 
a purpose rather than because it is something that is good to do. 
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Niemi and Darrell Anderson. 1970. $2.00, with ERIC/AE. 
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